v0.8.8 dev to main - #65

The Phase 2 plan existed only in conversation and was lost to a context
compaction, leaving two comments in store.js pointing at a design nobody could
read. This writes it down as a specification.

The substance: panel *settings* become per-panel with a global link toggle
defaulting to linked, rather than global-plus-overrides (two sources of truth for
every value, and no clean answer for what a slider shows) or always-independent
(breaks the default case and reintroduces the drift linkColorScale exists to
prevent). The push-to-other-panel button that prompted this work falls out of
per-panel state as an object copy.

Surface is measured rather than guessed: 4 files, 17 bare useStore() destructure
sites, 25 selector calls, 19 LayerPanel sections — and notably 0 data hooks,
since they take settings as props from ViewerPanel rather than reading the store.
Staged 2a–2e so the large mechanical migration lands while behaviour is still
frozen and any regression is unambiguous.

Also recorded, both found while checking the plan against the code:

- CLAUDE.md described edgeFile as global in three places. Phase 1 moved it to
  panels[i].edgeFile and left no global behind, so the docs contradicted the
  code. Corrected here rather than deferred — a stale architecture note is worse
  than a missing one.

- regions and measurements carry no panelIndex and Viewer renders them
  unfiltered, so every region draws in both panels at identical image-pixel
  coordinates. Harmless with one dataset in both panels, wrong with two: a
  polygon on a 6.5 mm Visium capture area redraws at those coordinates on a
  55 µm seqFISH ROI. Logged as a Phase 1 bug to fix separately, not folded into
  this plan.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Split screen shipped with `regions` and `measurements` still global and carrying
no panel index. The visible symptom is that every annotation draws in both
panels at identical image-pixel coordinates — a polygon over a 6.5 mm Visium
capture area reappearing over a 55 µm seqFISH ROI, where it means nothing.

Two consequences were worse, because they produce confident wrong answers
rather than obvious breakage:

- CSV export read `region.panelIndex ?? 0` while nothing ever *wrote*
  panelIndex, so every export resolved against panel 0's dataset. Exporting a
  region drawn in panel 1 posted panel 1's cell ids to panel 0's endpoint. The
  read side was written for a field the write side never set.

- Measurement labels compute `distPx * pixelSize` using the *rendering* panel's
  pixel size, so one measurement read as two different distances. Verified on
  CosMx (1.0 µm/px) beside MERSCOPE (0.108 µm/px): the same 100 px line reads
  100.0 µm and 10.8 µm.

Every annotation now carries `panelIndex`, stamped at creation; ViewerPanel
renders only its own; the in-progress polygon tracks which panel is drawing so
its dashed outline does not shadow the other; and clearAnnotations is scoped,
since the Clear button lives in each panel's own toolbar. Omitting the index
still clears everything. Anything created before this has no panelIndex and is
read as panel 0, the only place it could have come from.

The sidebar region list is shared, so in split mode each entry is labelled with
its panel — a bare cell count does not say which tissue it came from when the
panels hold different datasets.

Adds Vitest and the repo's first frontend tests (10, in
src/store.annotations.test.js), written failing before the fix. Store logic is
plain JS, so these need no DOM and no jsdom. This is a small down payment on the
risk named in docs/split_screen_phase2.md: that plan migrates ~30 state keys
across 19 components with no automated check behind it.

Verified in a two-dataset split: annotation scope 0/1 per panel, export resolves
to merscope-vpt-smallset rather than cosmx-mousebrain, and the measurement label
renders only in its own panel.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

… 2a)

First stage of docs/split_screen_phase2.md, and deliberately behaviour-frozen:
the ~30 display settings move off the store root onto each panel, but every
write still lands on every panel, so one sidebar drives both exactly as before.
Landing the large mechanical diff while behaviour is fixed means any regression
is unambiguously a bug in the move rather than a disagreement about intent.

- makeSettings() builds the defaults. A factory rather than a constant because
  `layers` and `hiddenLrms` are containers; one shared object would alias the
  panels, which is the exact bug the structure exists to prevent.
- patchSettings(patch, panelIndex = null) is the only writer. A null index
  writes to all panels — the 2a behaviour — and is the single place where 2b's
  link toggle will decide "one panel or all". The named setters are kept as thin
  wrappers so no write call site had to change.
- usePanelSettings() returns the store merged with one panel's settings, with
  the panel coming from PanelIndexContext or passed explicitly by ViewerPanel.
  That shape makes the 17 bare useStore() sites a one-identifier swap with their
  destructuring untouched.
- The data hooks are unchanged, as the plan predicted: they take settings as
  props from ViewerPanel rather than reading the store.

The one real regression risk was setPanelDataset. It rebuilt the panel with
makePanel(), which now carries fresh settings — that would have wiped edge
width, palettes and layer visibility, none of which a dataset change has ever
reset. It now applies a named RESET patch and keeps the rest, with a test
pinning both halves.

Verified behaviour-preserving by capturing all 28 effective setting values from
the running app before the change and diffing after: no differences. Also
checked live on two panels holding different datasets (cosmx-mousebrain,
merscope-vpt-smallset) that layer toggles, edge width, palette and colour-by all
still move in lockstep, with a clean console in both the dev server and the
production container.

Adds store.settings.test.js (13 tests); 23 frontend tests total.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Second stage of docs/split_screen_phase2.md. The sidebar gains tabs and a link
checkbox, so the two panels can be styled independently — two renderings of one
dataset side by side (cluster colouring against gene expression, two palettes,
two filters) was previously impossible, since every setting was one value.

Linked stays the default, which is the pre-2b behaviour exactly. That is the
same figure-integrity argument as linkColorScale: two panels that each drifted
to their own palette and clamp look comparable and are not.

- activePanel and linkSettings live in the store. patchSettings resolves the
  write target — all panels when linked, the active tab when not — and is the
  only place that decision is made, so no setter knows about tabs.
- getSetting reads the *active* panel. Read-modify-write setters depend on this:
  unlinked, toggleLrm must build its next Set from the panel it is about to
  write, not panel 0, or it drops whatever that panel had hidden.
- Re-linking adopts the active panel's settings rather than merely resuming
  propagation. Leaving the divergence in place would put a control labelled
  "linked" over two visibly different panels and converge them only partially on
  the next edit. cloneSettings deep-copies the containers; a shallow copy would
  leave the panels aliasing, so the next unlinked edit would write to both.
- CellInfoPanel now reads the clicked panel's settings rather than the active
  tab's, and AnnotationToolbar is bound to its own panel.

Also folds in 2d, which the plan had sequenced last. That ordering was wrong:
2b promises "editing panel 1 only", and a global dataset reset breaks it — an
action on panel 2 still wipes panel 1's filter, so 2b alone would ship a control
that lies. The panel whose dataset or edge file changed is always reset; the
others only while linked, where they share one set of values and a stale filter
would 400 on every viewport change.

Verified live on two panels over one dataset: viridis and inferno rendering side
by side with transcripts on in one panel only, re-link collapsing both to the
active tab's palette with no shared containers, and — unlinked — panel 1 keeping
its colour-by across a panel 2 dataset change while panel 2 is cleared. Linked,
both still reset. Clean console in the dev server and the production container.

11 new tests; 33 frontend tests total. The fixture now resets activePanel and
linkSettings between tests — they are what the targeting rule reads, so leaving
them set silently redirected a later test's writes to the wrong panel.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…y dataset

Two things, found together: 2c is the last functional stage of
docs/split_screen_phase2.md, and verifying it surfaced a regression I shipped in
ab9d739 that had broken every /edges endpoint.

## The regression

`DUCKDB_MEMORY_LIMIT` lost its fixed 8GB default in v0.8.4 so the cap could be
sized from memory actually available, and both compose files now pass it through
empty. But `edge_reader.py` kept its own connection setup reading
`os.getenv("DUCKDB_MEMORY_LIMIT", "8GB")` — and that default applies only when
the variable is *absent*. Set-but-empty yields "", so DuckDB got
`SET memory_limit=''` and raised

    ParserException: Parser Error: Memory limit must have a number

on every edge query, in the dev server and the container alike. Edges, the tissue
graph and the LRM catalogue were dead on all nine datasets. It did not show up
earlier because the failure needs the variable *set to empty*, which only the
compose files do — a bare `python3 tests/golden_snapshot.py` inherits an unset
variable and passes.

The real defect is two modules defaulting one environment variable two different
ways, so EdgeReader now goes through `duck.connect()`. It also picks up the
temp_directory it never had, so a large edge query can spill rather than fail.
`duck.py` additionally strips the value, since a whitespace-only setting is
truthy and would reach DuckDB unchanged — a hole the new check found.

`backend/tests/duckdb_config_check.py` covers unset / empty / blank / explicit
against a real edge query. The golden snapshot structurally cannot catch this.

## 2c

`pushSettings(from, to, allowed)` copies one panel's settings onto the other,
sanitised for the target. The button sits under the link toggle, unlinked only,
labelled with its direction.

The guard is the substance: a filter naming a column the target lacks 400s on
every viewport change and the panel silently stops rendering, so those settings
are dropped rather than copied. Column names come from /cells/schema and
/edges/schema, which the store never fetches — the component gathers the
vocabulary so the store action stays pure and testable. A gene allowlist with no
overlap becomes null ("no filter") rather than an empty Set ("show no species"),
matching a dataset change; colour clamps reset across different datasets, since
[0, 4000] onto data topping out at 70 paints everything one colour.

Verified across MERSCOPE → CosMx (zero gene overlap) through the real button:
geometry and palette copied, gene allowlist and clamp dropped, source untouched,
target still rendering 5000/38996 cells. 46 frontend tests, golden guard 238/238.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Last stage of docs/split_screen_phase2.md, which is now closed out.

The hosted manual needed correcting rather than extending. It said "Both panels
share all layer settings (visibility, color-by, LRM filter, etc.) but have
independent pan and zoom positions" — untrue since Phase 1 gave each panel its
own dataset, and wrong through two releases. Someone reading it would not have
known the feature they were looking for existed.

The Split-Screen section now covers two datasets side by side, the panel tabs
and link checkbox, the copy button and what it declines to copy, micron-based
zoom matching, and the shared colour scale. Two points get called out as notes
because they are surprising rather than discoverable: re-linking re-syncs to the
tab you are on, so pick that tab first; and Match works in microns, which is
what makes 20% of a 6.5 mm Visium capture area comparable to 20% of a 55 µm
seqFISH region rather than fifty-fold apart.

Also corrected nearby claims that the same work invalidated: Clear removes that
panel's annotations rather than everything, annotations belong to the panel that
drew them, and the dataset/image/edge pickers move into the panel headers in
split mode. The dataset-picker section gained a note on what a dataset change
clears and what it keeps, since that surprised me while writing the tests.

Verified rendered in a browser rather than by reading the source: all five
headings present, the old claim gone, anchors resolve, HTML balanced.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Fix edges being dead in v0.8.4, and finish split-screen per-panel settings

Writes up #59 before touching code, as with split-screen Phase 2. Three lab
directives shape it beyond the issue text, and two of them change current
behaviour:

- The tissue graph is ground truth: shown or hidden, never subset by a filter.
  Today one useEdges request feeds both the graph and the edge layer, so
  filtering edges thins the structural graph too.
- Cell and edge filtration are completely independent. Today
  query_grouped(cell_ids=…) emits `sending IN S AND receiving IN S`, so filtering
  cells removes edges.
- Density is applied last, being a rendering control rather than a selection.
  This one already holds — the grouped query samples an outer select wrapping the
  filtered, grouped subquery — and the plan records it as an invariant to
  preserve rather than work to do.

Findings that shaped the design, verified rather than assumed:

- Half of #59 already ships. sending_type/receiving_type are populated columns
  and are already offered in the edge filter dropdown; on mouse_ileum_tiny,
  sending_type=Fibroblast gives 55 of 223 edges with one sending type and all
  five receiving types. The blocker is that edgeFilter holds exactly one filter —
  the composition gap deferred in #45 — not the sending/receiving distinction.
- The tissue graph needs its own fetch rather than a passes_filter flag on the
  shared result, and density-last is why: the two layers want opposite orderings
  of the same pipeline, and a flag would be read after sampling had already
  thinned the rows it describes.
- Efficiency, which the issue asks about directly: one hash semi-join becomes
  two, on a query that already performs one. filter_cell_ids() and
  duck.register_ids() already exist.

Sizing measured across the bundled datasets — 169,219 unique edges on
cosmx-mousebrain is the largest. Rendering that is not the constraint; the ~15
fields per edge in the payload are, hence a lean projection staged as optional
work to be measured first.

CLAUDE.md indexes the plan and flags the both-endpoints rule as slated for
reversal, keeping the original reasoning since it still explains the code.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…ound it

I claimed "half of #59 already ships", demonstrating a sending_type filter
returning 55 of 223 edges on mouse_ileum_tiny. The mechanism works, but the
demonstration was against invented data and the claim was wrong in substance.

Tracing the column:

- sample_data/make_edges.py writes it, and its own docstring says
  `cell type (simulated)` — the tidy Endothelial/Immune/Fibroblast labels in
  every bundled fixture are the generator's invention.
- Of the six export scripts in r/, only niches_xenium.R can populate it, and
  only when the user passes --celltype. niches_cosmx, merscope, seqfish, visium
  and visium_hd all pass celltype.col = NULL, so on those platforms the column
  does not exist.

So on real output it is absent five times out of six, opt-in on the sixth, and
carries one label where the issue asks for any cell metadata column —
mouse_ileum_tiny's cells table has cluster, region, pseudotime and
seurat_clusters, none of which is in the edge file.

The plan now resolves both dropdowns through the cells table uniformly, with no
fast path: a fast path for the one case where the column happens to exist would
mean two code paths answering the same question differently depending on which
ran, and the frozen-at-scoring-time value can disagree with a re-annotated cells
table. The testing section no longer leans on the simulated column either.

Whether the column should exist at all is raised as an open question rather than
answered — it is a NICHESv2-side decision, and whether it is duplication or
provenance depends on whether NICHESv2 used the label when scoring, which is not
answerable from this repo.

docs/data_format.md and CLAUDE.md both presented it as a plain optional field,
which reads as "usually there". Both now say what populates it and that it
should not be relied on.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…ssue graph (#59)

Implements docs/edge_filter_independence.md. The pipeline is now explicit, and
the order is the contract:

    all edges in viewport
      → density filter        deterministic, spatially random
      → EDGESET A             → tissue-graph layer   (query_structure)
      → sending filter
      → receiving filter
      → edge-table filters
      → EDGESET B             → edge-data layer      (query_grouped)

Three lab directives drove it, two of which reversed existing behaviour.

The tissue graph is ground truth. It gets its own query, and query_structure
takes no filter arguments at all — not "they default to none", but no parameter
to pass, so nothing can wire one in later. It is a separate request rather than
a flag on the shared one because the two layers want opposite things from
sampling, and a passes_filter column would be read after the sample had already
thinned the rows it describes. The projection is lean (edge + four coordinates),
measured 2.2–2.4x smaller than the grouped payload.

Cell and edge filtering are independent. cell_filter no longer reaches the edge
request at all; sending_ids and receiving_ids constrain the two endpoints
separately, so both set gives the intersection and one set leaves the other end
free. An edge may now terminate on a cell that is not drawn, which is intended.
Both resolve from the *cells table* rather than the edge file's own
sending_type/receiving_type — those are absent on five of six platforms as the
r/ scripts stand, carry one label where any cell column is wanted, and are
frozen at scoring time.

Density is one slider over both layers. A second slider was built and removed:
the argument for it does not hold, since unfiltered the two layers draw exactly
the same number of lines and the graph already has an opacity control, which is
the better lever for clutter.

The real bug in the first cut was that the two queries sampled independently.
Two bernoulli draws at 10% overlap only ~1%, so edge data appeared where the
graph beneath it had been sampled away. density_predicate replaces USING SAMPLE
with a deterministic hash of the edge id: the same edge gets the same verdict in
every query, so the predicate commutes with the filters and B is a subset of A
at every density. It is also stable across re-fetches, where bernoulli flickered
on each pan.

Also: edge_filter becomes edge_filters, a list and-ed together, closing the
composition gap deferred in #45. The old cell_filter and edge_filter fields are
still accepted so an older frontend against a newer backend keeps working.

Verified rather than assumed:
- The intersection matches ground truth computed independently from the parquet
  and the metadata CSV — a full 3x3 sending/receiving matrix, zero mismatches.
- backend/tests/edge_pipeline_check.py asserts the graph is unmoved by any
  filter, that edge data is always a subset of it, and that sampling is stable:
  9 datasets across 6 platforms, at densities 1.0 / 0.5 / 0.1.
- Visually on Xenium: applying a sending filter leaves the grey graph pixel-for-
  pixel identical while the coloured edge layer thins; at 15% density every
  coloured edge sits on a grey line.
- Split screen with two datasets and per-panel endpoint filters, clean console.
- Golden snapshot 249 probes, frontend 52 tests, duckdb config check.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Independent sending/receiving edge filters, and unfilter the tissue graph

Click a cell, press "show local neighbourhood", and see what it is connected to
in the tissue graph — drawn on the canvas and summarised in the info panel:
neighbour and edge counts, the enclosing radius in µm, composition by any cell
metadata column, and the strongest LRMs across its incident edges.

The design note is docs/neighborhood_summary.md.

Computed server-side, and that is the point rather than an implementation
detail. The obvious version filters the `edges` array the frontend already
holds; it would appear to work and be wrong twice over, since that array is
density-sampled (nine of ten neighbours missing at the default) and
viewport-bounded (a neighbour just off-screen does not exist, and the answer
changes as you pan). A neighbourhood is a property of the tissue, not of the
current view, so the query ignores density, the viewport, the endpoint filters
and the LRM checklist. Cost was measured first and is not a reason to avoid it:
15 ms on the 3.8M-row CosMx file, no index, no cache.

Two marks are drawn because the radius alone would mislead. Connectivity is
anisotropic — a cell at a tissue boundary has neighbours on one side only — so a
disc around it encloses many cells it is not connected to. The highlighted cells
are the honest answer; the circle is the spatial scale the issue asked for.

Composition resolves against the cells table rather than the edge file's
sending_type, for the reasons recorded in v0.8.6: that column is absent on five
of six platforms, carries one label, and is frozen at scoring time. It keys on
the frame's cell_id *column* — the frame carries a plain RangeIndex, and my
first version indexed by position, which matched nothing and reported every
neighbour as missing.

Also fixed, found while testing this: CellInfoPanel rendered stale `detail` for
one frame after the selection cleared, because `detail` is state and outlives
`selectedCell`. The new section dereferenced it and took the whole app to the
error boundary on a dataset change. Guarded on both.

Verified:
- Counts and radius match ground truth computed independently from the parquet,
  on all 9 bundled datasets across 6 platforms; 3–125 ms each.
- End to end through the real button on Xenium: 11 neighbours, 18 edges, 12.8 µm,
  the highlight landing exactly on the tissue graph vertices joined to the
  clicked cell — and visibly enclosing unconnected cells the circle would have
  claimed.
- 56 frontend tests, including that the highlight carries its panel and is
  dropped on selection change or a dataset change in its own panel.
- Golden snapshot 249 probes, edge pipeline check, DuckDB config check.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Local neighbourhood highlighting and summaries

…sion in README

Three tidy-ups, plus a real hole found in my own test while doing them.

Empty metadata columns. `fov` and `transcript_count` are entirely null on the
bundled MERSCOPE dataset, but color-values reported them as a continuous 0–0
range. The UI then offered a range slider that did nothing, a legend with no
span, and a filter that correctly matched no cells while looking broken. They now
come back with `empty: True` and the filter section says "no values in this
column". `type` is still set so nothing switching on categorical-vs-continuous
needs a third case, and the cross-panel merge calls a column empty only when it
is empty in every panel.

The pan re-render storm. `usePanelSettings` subscribed to the whole store, so
every sidebar section re-rendered on each OpenSeadragon viewport-change event.
It now ignores `viewports` and `viewportActual`, which are the only continuously
rewritten keys and which no consumer of that hook reads — ViewerPanel takes
`viewports[panelIndex]` through its own selector and Match zoom reads
`viewportActual` via getState(). Measured over one simulated pan of 120 writes:
121 re-renders before, 0 after, with ordinary settings changes still delivered.
Narrowing the rest means a selector per section and is not worth it without
component tests.

Version in the README, and the bump list recorded in CLAUDE.md: three files move
together — package.json, main.py, README — while every other v0.x.y in the tree
is a historical reference that must not be swept along. Audited: the two
authoritative files agreed, and all other mentions were correctly historical.

The hole. tests/edge_pipeline_check.py called
`color_values("metadata", None, field)`, but the signature is
`(mode, field, genes, categorical)` — so the field went in as `genes`, the call
returned the empty result, `spec_for` returned None, and the filtered-subset
assertion was silently skipped on every dataset. The check printed OK while not
testing its own core property. Fixed, and it now says so out loud when no filter
resolves rather than passing over it. With the assertion actually running the
property still holds, on all 9 datasets and 3 densities — so the earlier
hand-verification was right, but the automated claim was weaker than stated.

Golden snapshot 249 probes, edge pipeline check, DuckDB config check, 56
frontend tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Report empty metadata columns, stop the pan re-render storm, version in the README